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1 . \A method of processing image data defining a 
plurality of sequences of images, each from a respective 
camera A of a plurality of objects moving in a scene to 
produce signals defining representations of the objects 
in a traree-dimensional computer model, the method 
compr i s ing^ 

processing image data from a first of the cameras 
to identify image data relating to objects in the scene; 

processing image data from a second of the cameras 
to identify image data relating to objects in the scene; 

processing Vthe identified image data from the first 
camera for each object to define an object representation 
in a modelling space having a height dependent upon the 
image data for the \ph ject from the first camera; 

processing the identified image data from the second 
camera for each ob jecA to define an object representation 
in the modelling space Viaving a height dependent upon the 
image data for the object from the second camera; 

comparing the height of the representation of each 
object generated in dependence upon image data from the 
first camera with the height of the representation of the 
corresponding object generated in dependence upon image 
data from the second camera; ^ind 

generating object representations in the three- 
dimensional computer model in dependence upon the height 
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2. A method according to claim 1, wherein the modelling 
space in which the object representations are defined 

5 using image data from the first camera and image data 
from the second camera is the three-dimensional computer 
model . 

3. A method according to claim 2, wherein the step of 
10 generating object representations in the three- 
dimensional computer model comprises modifying the taller 
representation when the heights of corresponding 
representations are not within a predetermined amount of 
each other. 
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3 4. A method according to claim 3, wherein, when the 

0 

heights of corresponding representations are not within 
the predetermined amount of each other, the taller 
representation is^ modified to give a representation 
20 having a height based on the height of the smaller 
representation . 

5. A method according to claim 3, wherein, when the 
heights of the corresponding representations are not 
25 within the predetermined amount of each other, a further 
representation is defined in the three-dimensional model 
using part of the image data from which the taller 
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representation was defined. 

6. A method according to claim 5, wherein, when the 
heights of the corresponding representations are not 

5 within the predetermined amount of each other, the taller 
representation is split into a first portion having a 
height corresponding to the height of the smaller 
representation and a second portion comprising the 
remaining part of the taller representation, and wherein 
10 the further representation is defined by re-positioning 
the second portion in the three-dimensional model, 

7. A method according to claim 6, wherein the second 
portion is repositioned in dependence upon a 

15 representation defined on the basis of image data from 
the camera which produced the smaller representation. 

8. A method according to claim 7, wherein the second 
portion is re-positioned by: 

20 identifying which of the representations defined on 

the basis of image data from the camera which produced 
the smaller representation overlaps the image data used 
to define the taller representation in the image space 
of the camera which produced the taller representation; 

25 and 

re-positioning the second portion in dependence upon 
the position of the identified representation in the 
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three-dimensional model, 

9. A method according to claim 8, wherein the second 
portion is re-positioned by: 

5 mapping at least part of each representation defined 

on the basis of image data from the camera which produced 
the smaller representation from the three-dimensional 
model to the image space of the camera which produced the 
taller representation; 

0 determining which projected representation overlaps 

the image data for the taller representation in the image 
space of the camera which produced the taller 
representation ; and 

re-positioning the second portion in dependence upon 

5 the position in the three-dimensional model of the 
representation which, when projected into the image space 
of the camera which produced the taller representation, 
overlapped the image data for the taller representation. 

0 10. A method according to claim 9, wherein the second 
portion is re-positioned so that the centre of its base 
is at the same position as the centre of the base of the 
representation which overlapped the image data for the 
taller representation . 

5 

11. A method according to claim 1, wherein each object 
representation is ofefined as a planar surface with its 
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base on a predetermined surface in the modelling space 
and with a position and size in dependence upon a polygon 
bounding the image \data for the object. 

12. A method according to claim 11, wherein the polygon 
is a rectangle . 

13. A method according to claim 12, wherein the sides 
of the rectangle are parallel to the sides of the image. 

14. A method according to claim 11, wherein the width 
of the planar surface is determined by the width of the 
bounding polygon in the image data, and the height of the 
planar surface is calculated using the aspect ratio of 
the bounding polygon in the image data. 

15. A method according to claim 11, wherein the planar 
surface lies within a vertical plane. 

16. A method according to claim 1, further comprising 
the step of generating image data by rendering an image 
of the three-dimensional computer model in which texture 
data based on the processed image data is rendered onto 
the representation of each object. 

17. A method according to claim 16, further comprising 
the step of generating a signal conveying the image data. 
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18. A method according to claim 17, further comprising 
the step of recording the signal. 

19. A method according to claim 16, further comprising 
5 the step of displaying an image of the objects using the 

generated image data . 

20. A method according to claim 16, further comprising 
the step of making a recording of the image data either 

10 directly or indirectly. 

21. A method\of image processing in which image data 
from first and ^second cameras is processed to identify 
image data relating to respective objects, the height of 

15 each object in a modelling space is determined using the 
identified image oata, and the heights of objects 
determined using image data from the first camera are 
compared with the heights of objects determined using 
image data from the second camera to determine which if 

20 any identified image data relates to more than one 
object . 

22. An image processing method in which image data from 
a first camera of objects in a scene is processed to 

25 identify image data relating to respective objects, and 
image data from a second camera of the objects in the 
scene is processed to determine whether any of the 
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identified image data from the first camera relates to 
more than one object by comparing a size parameter of 
each object determined from the image data of the first 
camera with the corresponding size parameter determined 
5 from the image data of the second camera. 

23- An iNmage processing apparatus for processing image 

data defirving a plurality of sequences of images, each 

from a respective camera, of a plurality of objects 
10 moving in >a scene to produce signals defining 

representations of the objects in a three-dimensional 

computer model , Ythe apparatus comprising: 

means for processing image data from a first of the 

cameras to identify image data relating to objects in the 
15 scene; \ 

means for processing image data from a second of the 

cameras to identify image data relating to objects in the 

scene; \ 

means for processing; the identified image data from 
20 the first camera for each object to define an object 

representation in a modelling space having a height 

dependent upon the image da^a for the object from the f 

first camera; \ 

means for processing the identified image data from 
25 the second camera for each object to define an object 

representation in the modelling sj&ace having a height 

dependent upon the image data for tthe object from the 
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second camera ; 

means for comparing the height of the representation 
of each object generated in dependence upon image data 
from the first camera with the height of the 
representation of the corresponding object generated in 
dependence upon image data from the second camera; and 

means for generating object representations in the 
three-dimensional computer model in dependence upon the 
height comparisons . 

24. Apparatus according to claim 23, wherein the 
modelling space in which the object representations are 
defined using image data from the first camera and image 
data from the second camera is the three-dimensional 
computer model . 

25. Apparatus according to claim 24, wherein the means 
for generating object representations in the three- 
dimensional computer model comprises means arranged to 
modify the taller representation when the heights of 
corresponding representations are not within a 
predetermined amount of each other. 

26. Apparatus according to claim 25 f arranged to perform 
processing such that, when the heights of corresponding 
representations are not within the predetermined amount 
of each other, the taller representation is modified to 
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give a representation having a height based on the height 
of the smaller representation. 

27. Apparatus according to claim 25, arranged to perform 
5 processing such that, when the heights of the 

corresponding representations are not within the 
predetermined amount of each other, a further 
representation is defined in the three-dimensional model 
using part of the image data from which the taller 
10 representation was defined. 

28. Apparatus according to claim 27, arranged to perform 
processing such that, when the heights of the 
corresponding representations are not within the 

15 predetermined amount of each other, the taller 
representation is split into a first portion having a 
height corresponding to the height of the smaller 
representation and a second portion comprising the 
remaining part of the taller representation, and wherein 

20 the further representation is defined by re-positioning 
the second portion in the three-dimensional model. 

29. Apparatus according to claim 28, arranged to perform 
processing such that the second portion is re-positioned 

25 in dependence upon aXrepresentation defined on the basis 
of image data from theXcamera which produced the smaller 
representation . \ 
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30. Apparatus according to claim 29, arranged to perform 
processing such that the second portion is re-positioned 
by: 

identifying which of the representations defined on 
5 the basis of image data from the camera which produced 
the smaller representation overlaps the image data used 
to define the taller representation in the image space 
of the camera which produced the taller representation; 
and 

10 re-positioning the second portion in dependence upon 

the position of the identified representation in the 
three-dimensional model . 

31. Apparatus according to claim 30, arranged to perform 
processing such that the second portion is re-positioned 
by: 

mapping at least part of each representation defined 
on the basis of image data from the camera which produced 
the smaller representation from the three-dimensional 
model to the image space of the camera which produced the 
taller representation; 

determining which projected representation overlaps 
the image data for the taller representation in the image 
space of the camera which produced the taller 
representation ; and 

re-positioning the second portion in dependence upon 
the position in the three-dimensional model of the 
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representation which, when projected into the image space 
of the camera which produced the taller representation, 
overlapped the image data for the taller representation, 

32. Apparatus according to claim 31, arranged to perform 
processing such that the second portion is re-positioned 
so that the centre of its base is at the same position 
as the centre of the base of the representation which 
overlapped the image data for the taller representation. 

33. Apparatus, according to claim 23, arranged to perform 
processing sucn\ that each object representation is 
defined as a planar surface with its base on a 
predetermined surface^ in the modelling space and with a 
position and size in dependence upon a polygon bounding 
the image data for the object. 

34. Apparatus according to claim 33, wherein the polygon 
is a rectangle . 

35. Apparatus according to claim 34, wherein the sides 
of the rectangle are parallel to the sides of the image. 

36. Apparatus according to claim 33, arranged to perform 
processing such that the width of the planar surface is 
determined by the width of the bounding polygon in the 
image data, and the height of the planar surface is 
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calculated using the aspect ratio of the bounding polygon 
in the image data . 

37. Apparatus according to claim 33 , arranged to perform 
processing such that the planar surface lies within a 
vertical plane. 

38. Apparatus according to claims 23, further comprising 
means for generating image data by rendering an image of 
the three-dimensional computer model in which texture 
data based on the processed image data is rendered onto 
the representation of each object . 

39. Apparatus according to claim 38 further comprising 
means for displaying an image of the objects using the 
generated image data . 

40. An image processing apparatus operable to process 
image data rk)m first and second cameras to identify 
image data relatsing to respective objects, to determine 
the height of eacnSobject in a modelling space using the 
identified image data, and to compare the heights of 
objects determined using\image data from the first camera 
with the heights of objects determined using image data 
from the second camera tos. determine which if any 
identified image data relates ro more than one object. 
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41. An image processing apparatus operable to process 
image data from a first camera of objects in a scene to 
identify image data relating to respective objects, and 
to process image data from a second camera of the objects 
5 in the scene to determine whether any of the identified 
image data from the first camera relates to more than one 
object by comparing a size parameter of each object 
determined from the image data of the first camera with 
the corresponding size parameter determined from the 
10 image data of the second camera. 

42Vv A storage medium storing instructions for causing 
a programmable processing apparatus to perform a method 
accordingN^o - any of . claim^ 1 to 22 ". 

15 

43. A signal ctemveying instructions for causing a 
programmable processing apparatus to perform a method 
^ according to -any - o dE cla^k 1 to 22 . 

20 44. A method of processing image data defining a 
plurality of sequences of images, each from a respective 
camera, of an object moving in a scene to produce signals 
defining a representation of the object in a three- 
dimensional computer model, the method comprising: 

25 processing image data from a first of the cameras 

to identify image data relating to the object in the 
scene; 
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processing image data from a second of the cameras 
to identify image data relating to the object in the 
scene; 

applying a transformation to the identified image 
5 data from the first camera which defines a mapping from 
the ground plane in the space of the image data of the 
first camera to a surface in a modelling space; 

applying a transformation to the identified image 
data from the second camera which defines a mapping from 
10 the ground plane in the space of the image data of the 
second camera to the surface in the modelling space; 

comparing the transformed image data from the first 
and second cameras on the surface in the modelling space; 
determining which part of the image data represents 
15 shadow in dependence upon the comparison results; and 

generating a representation of at least the object 
in the three-dimensional model . 

45- A method according to claim 44, further comprising 
2 0 the step of generating a representation of the shadow in 
the three-dimensional model. 

46. A method according to claim 44 , wherein the surface 
in the modelling space is the ground plane in the three- 
25 dimensional model. 



47. A method according to claim 44, wherein it is 
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determined that aligned parts of the transformed image 
data represent shadow. 

48. A method according to claim 44, further comprising 
5 the step of generating image data by rendering an image 
of the three-dimensional computer model in which texture 
data based on the processed image data is rendered onto 
the representation of the object. 

10 49. A method according to claim 48, wherein the image 
data rendered onto' the representation is determined in 
dependence upon the comparison results . 

50. A method according to claim 48, further comprising 
15 the step of generating a signal conveying the image data. 

51. A method according to claim 50, further comprising 
the step of recording the signal. 

20 52. A method according to claim 48, further comprising 
the step of displaying an image of the object using the 
generated image data . 



25 



53. A method according to claim 48, further comprising 
the step of making a recording of the image data either 
directly or indirectly. 
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54. A method of generating a model of an object in a 
three-dimensional computer model by processing images of 
the object from a plurality of cameras, in which image 
data from a first camera is processed to identify image 
5 data relating to the object and its shadow together, and 
image data from a second camera is used to determine the 
identified image data from the first camera which relates 
to the shadow and the identified image data from the 
first camera which relates to the object. 

10 

^5. A method of generating a model of an object in a 
thBee-dimensional computer model, in which: 

\^ transformation is applied to image data from a 
first camera relating to the object and its shadow which 
15 maps the irr^ge data for one of the object and its shadow 
to a surface; \. 

a transformation is applied to image data from a 
second camera relatr*Kj to the object and its shadow which 
maps the image data fo\ one of the object and its shadow 
20 to the surface; and \. 

the object is modelleaSin dependence upon part of 
the transformed image data. \ 

56. Apparatus for processing image data defining a 
25 plurality of sequences of images, each from a respective 
camera, of an object moving in a scene to produce signals 
defining a representation of the object in a three- 



dimensional computer model, the apparatus comprising: 

means for processing image data from a first of the 
cameras to identify image data relating to the object in 
the scene; 

means for processing image data from a second of the 
cameras to identify image data relating to the object in 
the scene; 

means for applying a transformation to the 
identified image data from the first camera which defines 
a mapping from the ground plane in the space of the image 
data of the first camera to a surface in a modelling 
space; 

means for applying a transformation to the 
identified image data from the second camera which 
defines a mapping from the ground plane in the space of 
the image data of the second camera to the surface in the 
modelling space; 

means for comparing the transformed image data from 
the first and second cameras on the surface in the 
modelling space; 

means for determining which part of the image data 
represents shadow in dependence upon the comparison 
results; and 

means for generating a representation of at least 
the object in the three-dimensional model. 



57. Apparatus according to claim 56, further comprising 
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means for generating a representation of the shadow in 
the three-dimensional model. 

58. Apparatus according to claim 56 , wherein the surface 
5 in the modelling space is the ground plane in the three- 
dimensional model . 

59. Apparatus according to claim 56 , arranged such that 
^ it is determined that aligned parts of the transformed 
5ff 10 image data represent shadow. 

m 

4f 60. Apparatus according to claim 56, further comprising 

means for generating image data by rendering an image of 
^ the three-dimensional computer model in which texture 

H- 15 data based on the processed image data is rendered onto 
the representation of the object. 

61. Apparatus according to claim 60 , wherein the image 
data rendered onto the representation is determined in 

20 dependence upon the comparison results. 

62. Apparatus according to claim 60, further comprising 
means for displaying an image of the object using the 
generated image data . 

25 



63. Apparatus for generating a model of an object in a 
three-dimensional computer model by processing images of 
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the object from a plurality of cameras, the apparatus 
being operable to process image data from a first camera 
to identify image data relating to the object and its 
shadow together, and operable to use image data from a 
second camera to determine the identified image data from 
the first camera which relates to the shadow and the 
identified image data from the first camera which relates 
to the object . 

10 64. \&pparatus for generating a model of an object in a 
three-cHimensional computer model, comprising: 

meanss for applying a transformation to image data 
from a first camera relating to the object and its shadow 
which maps tfte image data for one of the object and its 
15 shadow to a surface; 

means for applying a transformation to image data 
from a second camera relating to the object and its 
shadow which maps thk image data for one of the object 
and its shadow to the surface; and 
20 means for modelling\the object in dependence upon 

part of the transformed image data. 

., . _ _ ,„ _ 

a programmable processing apparatus to perform a method 
\ 25 according to any of claim^ 44 - to 55\ 



66. A signal conveying instructions\ for causing a 
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programft^able processing apparatus to perform a method 
1 according N:o - any o f- claim^ 44 -to 55 . 



67. A method of processing image data defining a 
5 plurality of sequences of images, each from a respective 
camera, of an object moving in a scene to produce signals 
defining a representation of the object in a three- 
dimensional computer model, the method comprising: 

processing image data from a first of the cameras 
10 to identify image data relating to the object in the 
scene; 

processing image data from a second of the cameras 
to identify image data relating to the object in the 
scene ; 

15 processing the identified image data from the first 

camera and the identified image data from the second 
camera to determine a footprint of the object on the 
ground; and 

defining a model of the object in the three- 
20 dimensional computer model in dependence upon the 
determined footprint . 



68. A method according to claim 67, wherein the step of 
processing the identified image data to determine the 
25 footprint of the object on the ground comprises: 

applying a transformation to the identified image 
data from the first camera which defines a mapping from 
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the ground plane in the image data of the first camera 
to a surface in a modelling space; 

applying a transformation to the identified image 
data from the second camera which defines a mapping from 
the ground plane in the image data of the second camera 
to the surface in the modelling space; and 

comparing the transformed image data on the surface 
in the modelling space. 

69. A method according to claim 68, wherein the surface 
in the modelling space is the ground plane in the three- 
dimensional computer model. 

70. A method according to claim 68, wherein the outline 
of the image on the ground is determined in dependence 
upon the aligned and non-aligned portions of the 
transformed image data on the surface in the modelling 
space . 

71. A method according to claim 67, wherein, the step 
of defining the model of the object comprises defining 
the model using a plurality of vertical planar surfaces. 

72. A method according to claim 71, wherein the vertical 
planar surfaces are defined such that their bases 
approximate the outline of the object on the ground. 
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73. A method according to claim 71, wherein each planar 
surface is a rectangle. 

74. A method according to claim 71, wherein each planar 
surface is defined with a height determined in dependence 
upon the image data identified from the first camera or 
the image data identified from the second camera. 

75. A method according to claim 74, wherein the height 
of each planar surface is defined in dependence upon a 
rectangle bounding some or all of the image data relating 
to the object identified from the first camera or the 
second camera. 

76. A method according to claim 74, wherein each planar 
surface is defined to have the same height. 

77. A method according to claim 71, further comprising 
the step of generating a top for the model of the object 
in dependence upon upper edges of the vertical planar 
surfaces . 

78. A method according to claim 67, further comprising 
the step of generating image data by rendering an image 
of the modelled object, in which texture data based on 
the identified image data from at least one camera is 
rendered onto the model . 
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79. A method according to claim 7 8 , wherein each planar 
surface is mapped onto the image data of the first camera 
or the second camera, and the image data enclosed by each 
mapped surface is rendered onto the planar surface in the 
model . 

80. A method according to claim 78, further comprising 
the step of generating a signal conveying the image data. 

81. A method according to claim 80, further comprising 
the step of recording the signal. 

82. A method according to claim 78, further comprising 
the step of displaying an image of the object using the 
generated image data. 

83. A method according to claim 78, further comprising 
the step of making a recording of the image data either 
directly or indirectly. 

84. A method of generating a model of an object in a 
three-dimensional computer model by processing images of 
the object from a plurality of cameras, in which image 
data from a first camera is processed to identify image 
data relating to the object, image data from a second 
camera is used to determine which parts of the identified 
image data from the first camera relate to parts of the 
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object on or near the ground, and the object is 
represented in the computer model in dependence thereon . 

85. Apparatus for processing image data defining a 
5 plurality of sequences of images, each from a respective 

camera, of an object moving in a scene to produce signals 
defining a representation of the object in a three- 
dimensional computer model, the apparatus comprising: 
q means for processing image data from a first of the 

ftg 10 cameras to identify image data relating to the object in 

ay 

[yij the scene; 

Iq means for processing image data from a second of the 

^ cameras to identify image data relating to the object in 

the scene; 

f Z 15 means for processing the identified image data from 

™ the first camera and the identified image data from the 

second camera to determine a footprint of the object on 
the ground ; and 

means for defining a model of the object in the 
20 three-dimensional computer model in dependence upon the 
determined footprint . 

86. Apparatus according to claim 85, wherein the means 
for processing the identified image data to determine the 

25 footprint of the object on the ground comprises: 

means for applying a transformation to the 
identified image data from the first camera which defines 
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a mapping from the ground plane in the image data of the 
first camera to a surface in a modelling space; 

means for applying a transformation to the 
identified image data from the second camera which 
5 defines a mapping from the ground plane in the image data 
of the second camera to the surface in the modelling 
space; and 

means for comparing the transformed image data on 
fg the surface in the modelling space. 

10 

n\ 

[pi 87. Apparatus according to claim 86 , wherein the surface 

ff- 

"5 in the modelling space is the ground plane in the three- 

te dimensional computer model. 

= — i~ 

^ 15 88. Apparatus according to claim 86, arranged to 
^3 determine the outline of the image on the ground in 

dependence upon the aligned and non-aligned portions of 
the transformed image data on the surface in the 
modelling space. 

20 

89. Apparatus according to claim 85, wherein, the means 
for defining the model of the object comprises means for 
defining the model using a plurality of vertical planar 
surfaces . 

25 

90. Apparatus according to claim 89, arranged to perform 
processing such that the vertical planar surfaces are 
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defined such that their bases approximate the outline of 
the object on the ground. 

91. Apparatus according to claim 89, wherein each planar 
5 surface is a rectangle. 

92. Apparatus according to claim 89, arranged to perform 
processing such that each planar surface is defined with 
a height determined in dependence upon the image data 

10 identified from the first camera or the image data 
identified from the second camera. 

93. Apparatus according to claim 92, arranged to perform 
processing such that the height of each planar surface 

15 is defined in dependence upon a rectangle bounding some 
or all of the image data relating to the object 
identified from the first camera or the second camera. 

94. Apparatus according to claim 92, wherein each planar 
20 surface is defined to have the same height. 

95. Apparatus according to claim 89, further comprising 
means for generating a top for the model of the object 
in dependence upon upper edges of the vertical planar 

25 surfaces. 



96. Apparatus according to claim 85, further comprising 
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the step of generating image data by rendering an image 
of the modelled object, in which texture data based on 
the identified image data from at least one camera is 
rendered onto the model • 

5 

97. Apparatus according to claim 96, arranged to perform 
processing such that each planar surface is mapped onto 
the image data of the first camera or the second camera, 
and the image data enclosed by each mapped surface is 

10 rendered onto the planar surface in the model. 

98. Apparatus according to claim 96, further comprising 
means for displaying an image of the object using the 
generated image data . 

15 

99. Apparatus for generating a model of an object in a 
three-dimensional computer model by processing images of 
the object from a plurality of cameras, the apparatus 
being operable to process image data from a first camera 

20 to identify image data relating to the object, to use 
image data from a second camera to determine which parts 
of the identified image data from the first camera relate 
to parts of the object on or near the ground, and to 
represent the object in the computer model in dependence 

25 thereon. 

100. A stbrage medium storing instructions for causing 
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•a programmable processing apparatus to perform a method 
according to any of claim^ 67 - to 04 -. 

101. A srgnal conveying instructions for causing a 
programmable^rocessing apparatus to perform a method 
according to - any^oi? claim^ 67 -Lo 04 . 

102. A method of processing image data defining a 
sequence of images of a plurality of objects moving in 
a scene to produce signals defining representations of 
the objects in a three-dimensional computer model, and 
to generate image data by rendering an image of the 
three-dimensional computer model in accordance with a 
user-selected viewing direction, the method comprising: 

processing the image data to identify image data 
relating to respective objects in the scene; 

defining a representation of each object in the 
three-dimensional computer model, in dependence upon the 
identified image data; and 

generating image data by rendering an image of the 
three-dimensional computer model ^in accordance with a 
user-selected viewing direction, wherein, when the 
selected viewing direction is within a predetermined 
range of viewing directions, texture data based on the 
identified image data is rendered onto the object 
representations, and, when the selected viewing direction 
is not within the predetermined range of viewing 
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directions, a schematic of the positions of the objects 
in the scene is rendered. 

103. A method according to claim 102, wherein the 
representation of each object comprises a plurality of 
vertical planar surfaces alone. 

104. A method according to claim 102, wherein the 
representation of each object comprises a single vertical 
planar surface. 

105. A method according to claim 102, wherein the 
predetermined range of viewing directions is a range 
relative to a fixed direction in the computer model. 

106. A method according to claim 105, wherein the 
predetermined range of viewing direction is a range 
relative to the vertical direction in the computer model. 

107. A method according to claim 102, wherein the 
predetermined range of viewing directions is a range 
relative to the representation of an object. 

108. A method according to claim 107, wherein the 
representation of an object comprises at least one 
vertical planar surface, and the predetermined range of 
viewing directions is a range relative to a planar 
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surface . 

109. A method according to claim 102, wherein the 
schematic of the object positions is rendered from a 
predetermined viewing direction. 

110. A method according to claim 109, wherein the 
schematic is rendered from a vertical downward viewing 
direction . 

111. A method according to claim 102, further comprising 
the steps of processing the image data to determine at 
least one colour for each object, and generating image 
data to indicate the determined colour on the schematic 
of the object positions. 

112. A method according to claim 102, further comprising 
the step of generating a signal conveying the image data. 

113. A method according to claim 112, further comprising 
the step of recording the signal. 

114. A method according to claim 102, further comprising 
the step of displaying an image of the objects using the 
generated image data. 

115. A method according to claims 102, further comprising 
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the step of making a recording of the image data either 
directly or indirectly, 

116- A method of rendering an image in accordance with 
a user-selected viewing direction of a three-dimensional 
computer model comprising a representation and associated 
texture data for an object, the texture data being 
derived from image data recorded by at least one camera, 
the method comprising: 

rendering the texture data onto the representation 
for the object in accordance with the user-selected 
viewing direction when the user-selected viewing 
direction is within a predetermined range of viewing 
directions; and 

rendering a schematic of the positions of the object 
when the user-selected viewing direction is not within 
the predetermined range of viewing directions . 

3S17 . An image processing method in which object data 
defining a three-dimensional computer model of a 
pluralrt^ of objects in a scene is processed to generate 
image data^or an image of the objects and the scene such 
that : iC? 

in response t>o a first user input, the objects and 
the scene are rendered in accordance with a selected 
viewing direction; and 

in response to a seconfcL user input, image data is 
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rendered 



an image in which the positions of the 



objects in 



scene are represented. 



118. Apparatus for processing image data defining a 
sequence of images of a plurality of objects moving in 
a scene to produce signals defining representations of 
the objects in a three-dimensional computer model, and 
to generate image data by rendering an image of the 
three-dimensional computer model in accordance with a 
user-selected viewing direction, the apparatus 
comprising : 

means for processing the image data to identify 
image data relating to respective objects in the scene; 

means for defining a representation of each object 
in the three-dimensional computer model, in dependence 
upon the identified image data; and 

means for generating image data by rendering an 
image of the three-dimensional computer model in 
accordance with a user-selected viewing direction, 
operable such that, when the selected viewing direction 
is within a predetermined range of viewing directions, 
texture data based on the identified image data is 
rendered onto the object representations, and, when the 
selected viewing direction is not within the 
predetermined range of viewing directions , a schematic 
of the positions of the objects in the' scene is rendered. 



110 

119. Apparatus according to claim 118, operable to 
perform processing such that the representation of each 
object comprises a plurality of vertical planar surfaces 
alone . 

5 

120. Apparatus according to claim 118, operable to 
perform processing such that the representation of each 
object comprises a single vertical planar surface. 

121. Apparatus according to claim 118, operable to 
perform processing such that the predetermined range of 
viewing directions is a range relative to a fixed 
direction in the computer model. 

122. Apparatus according to claim 121, operable to 
perform processing such that the predetermined range of 
viewing direction is a range relative to the vertical 
direction in the computer model . 

20 123. Apparatus according to claim 118, operable to 
perform processing such that the predetermined range of 
viewing directions is a range relative to the 
representation of an object. 

25 124. Apparatus according to claim 123, operable to 
perform processing such that the representation of an 
object comprises at least one vertical planar surface, 
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and the predetermined range of viewing directions is a 
range relative to a planar surface. 

125. Apparatus according to claim 118, operable to 
5 perform processing such that the schematic of the object 

positions is rendered from a predetermined viewing 
direction . 

126. A method according to claim 125, operable to perform 
10 processing such that the schematic is rendered from a 

vertical downward viewing direction. 

127. Apparatus according to claim 118, further comprising 
means for processing the image data to determine at least 

15 one colour for each object, and means for generating 
image data to indicate the determined colour on the 
schematic of the object positions. 

128. Apparatus according to claim 118, further comprising 
20 means for displaying an image of the objects using the 

generated image data . 

129. Apparatus for rendering an image in accordance with 
a user-selected viewing direction of a three-dimensional 

25 computer model comprising a representation and associated 
texture data for an object, the texture data being 
derived from image data recorded by at least one camera, 



m 
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the apparatus comprising: 

means for rendering the texture data onto the 
representation for the object in accordance with the 
user-selected viewing direction when the user-selected 
5 viewing direction is within a predetermined range of 
viewing directions; and 

means for rendering a schematic of the positions of 
the object when the user-selected viewing direction is 
« not within the predetermined range of viewing directions . 

jjj ^s30 . An image processing apparatus operable to process 

3 obj&^t data defining a three-dimensional computer model 

l~ of a plurality of objects in a scene to generate image 

rf data for arKimage of the objects using first and second 

15 techniques suclJ^that: 

in the f irst\technique, the objects and the scene 



are rendered in accomance with a viewing direction; and 
in the second technique, image data is rendered for 



in which\>t 



a schematic image in which t>^ie positions of the objects 
20 in the scene are represented, 



^ — 1 r^l . A storage medium storing instructions for causing 

fX / 

^) ( a pr<^rammable processing apparatus to perform a method 

accordin^sto - any - of claim^ 102 -ku 117 

25 




132. A signal conveying instructions for causing a 
programmable processing apparatus to perform a method 
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(\ according to any -&€ claim^ 102 to 117 . 

133. A method of processing image data defining a 
plurality of sequences of images, each from a respective 

5 camera, of an object moving in a scene to produce signals 
defining a representation of the object in a three- 
dimensional computer model, the method comprising: 

processing image data from a first of the cameras 
f =~ to identify image data relating to the object in the 

55 10 scene; 

[S processing image data from a second of the cameras 

;J to identify image data relating to the object in the 

scene; 

s yj processing the identified image data from the first 

H; 15 camera and the identified image data from the second 
\Q camera to identify planar surfaces on which points on the 

object lie by matching feature points in the identified 
image data from the first camera with feature points in 
the identified image data from the second camera, and 
20 identifying planar surfaces on which matched feature 
points lie; and 

defining a model of the object in the three- 
dimensional computer model in dependence upon the 
identified planar surfaces. 

25 

134. A method according to claim 133, wherein corner 
points in the identified image data from the first camera 





114 

are matched with corner points in the identified image 
data from the second camera. 

135. A method according to claim 133, wherein the planar 
5 surfaces are identified by identifying planes on which 
the matched feature points lie, and determining 
boundaries of the planes using matched feature points 
which lie on more than one plane. 

10 136. A method according to claim 135, wherein each plane 
is identified by identifying a plane on which at least 
a predetermined number of feature points in the 
identified image data from the first camera lie, 
identifying the plane on which the matched feature points 

15 in the identified image data from the second camera lie, 
calculating a transformation between the plane in the 
image data from the first camera and the plane in the 
image data from the second camera, and testing the 
transformation using a plurality of other matched pairs 

20 of feature points. 

137. A method according to claim 136, wherein the 
predetermined number is four. 

25 138. A method according to claim 133, wherein the step 
of defining the model of the object comprises forming a 
model of planar surfaces in the three-dimensional 



computer model, each planar surface in the model 
corresponding to a planar surface identified in the image 
data from at least one of the cameras. 

139. A method according to claim 138, wherein the step 
of defining the model of the object comprises identifying 
a planar surface which touches the ground in the image 
data of a camera, defining a vertical planar surface in 
the three-dimensional computer model in dependence upon 
the identified planar surface which touches the ground, 
and defining a further planar surface in the three- 
dimensional computer model for each further planar 
surface in the image data of the camera such that the 
planar surfaces in the three-dimensional computer model 
and the image data have the same aspect ratio. 

140. A method according to claim 139, wherein a planar 
surface which touches the ground in the image data of the 
given camera is identified by: 

applying a transformation to the base corner points 
of planar surfaces in the image data from the first 
camera which defines a mapping from the ground plane in 
the image data of the first camera to a surface in a 
modelling space; 

applying a transformation to the base corner points 
of planar surfaces in the image data from the second 
camera which defines a mapping from the ground plane in 



the image data of the second camera to the surface in the 
modelling space; and 

comparing the transformed corner points to determine 
which ones lie within a predetermined distance of each 
other, 

141. A method according to claim 140 , wherein the surface 
in the modelling space is the ground plane in the three- 
dimensional computer model, 

142. A method according to claim 141, wherein the defined 
vertical planar surface in the three-dimensional computer 
model is defined with a base defined by transformed 
corner points from the given camera which lie within the 
predetermined distance of the corresponding transformed 
corner points from the other camera, and with an aspect 
ratio corresponding to the aspect ratio of the planar 
surface in the image data of the given camera to which 
the transformed corner points belong. 

143. A method according to claim 133, further comprising 
the step of generating image data by rendering an image 
of the modelled object, in which texture data based on 
the identified image data from at least one camera is 
rendered onto the model . 



144. A method according to claim 143, wherein image data 



enclosed by each planar surface is rendered on the 
corresponding planar surface of the object model. 

145. A method according to claim 14 3, further comprising 
the step of generating a signal conveying the image data. 

146. A method according to claim 145, further comprising 
the step of recording the signal. 

147. A method according to claim 143, further comprising 
the step of displaying an image of the object using the 
generated image data . 

148. A method according to claim 143, further comprising 
the step of making a recording of the image data either 
directly or indirectly. 

149. A method of generating a model of an object in a 
three-dimensional computer model by processing images of 
the object from a plurality of cameras, in which image 
data from a first camera and a second camera is processed 
to match feature points in the image data from the first 
camera with feature points in image data from the second 
camera, the resulting matches are used to determine 
planar surfaces making up the object, and the object is 
represented in the computer model in dependence thereon. 
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150. Apparatus for processing image data defining a 
plurality of sequences of images, each from a respective 
camera, of an object moving in a scene to produce signals 
defining a representation of the object in a three- 
dimensional computer model, the apparatus comprising: 

means for processing image data from a first of the 
cameras to identify image data relating to the object in 
the scene; 

means for processing image data from a second of the 
cameras to identify image data relating to the object in 
the scene; 

means for processing the identified image data from 
the first camera and the identified image data from the 
second camera to identify planar surfaces on which points 
on the object lie, comprising means for matching feature 
points in the identified image data from the first camera 
with feature points in the identified image data from the 
second camera, and means for identifying planar surfaces 
on which matched feature points lie; and 

means for defining a model of the object in the 
three-dimensional computer model in dependence upon the 
identified planar surfaces. 

151. Apparatus according to claim 150, operable to 
perform processing such that corner points in the 
identified image data from the first camera are matched 
with corner points in the identified image data from the 
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second camera, 

152. Apparatus according to claim 150, operable to 
perform processing such that the planar surfaces are 

5 identified by identifying planes on which the matched 
feature points lie, and determining boundaries of the 
planes using matched feature points which lie on more 
than one plane. 

153. Apparatus according to claim 152, operable to 
perform processing such that each plane is identified by 
identifying a plane on which at least a predetermined 
number of feature points in the identified image data 
from the first camera lie, identifying the plane on which 
the matched feature points in the identified image data 
from the second camera lie, calculating a transformation 
between the plane in the image data from the first camera 
and the plane in the image data from the second camera, 
and testing the transformation using a plurality of other 
matched pairs of feature points . 

154. Apparatus according to claim 153, wherein the 
predetermined number is four. 



10 



?! 3 



15 



20 



25 



155. Apparatus according to claim 150, wherein the means 
for defining the model of the object comprises means for 
forming a model of planar surfaces in the three- 
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dimensional computer model, each planar surface in the 
model corresponding to a planar surface identified in the 
image data from at least one of the cameras. 

156. Apparatus according to claim 155, wherein the means 
for defining the model of the object comprises means for 
identifying a planar surface which touches the ground in 
the image data of a camera, means for defining a vertical 
planar surface in the three-dimensional computer model 
in dependence upon the identified planar surface which 
touches the ground, and means for defining a further 
planar surface in the three-dimensional computer model 
for each further planar surface in the image data of the 
camera such that the planar surfaces in the three- 
dimensional computer model and the image data have the 
same aspect ratio. 

157. Apparatus according to claim 156, operable to 
perform processing such that a planar surface which 

20 touches the ground in the image data of the given camera 
is identified by: 

applying a transformation to the base corner points 
of planar surfaces in the image data from the first 
camera which defines a mapping from the ground plane in 
25 the image data of the first camera to a surface in a 
modelling space; 

applying a transformation to the base corner points 
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of planar surfaces in the image data from the second 
camera which defines a mapping from the ground plane in 
the image data of the second camera to the surface in the 
modelling space; and 
5 comparing the transformed corner points to determine 

which ones lie within a predetermined distance of each 
other . 

158. Apparatus according to claim 157, wherein the 
10 surface in the modelling space is the ground plane in the 

three-dimensional computer model. 

159. Apparatus according to claim 158, operable to 
perform processing such that the defined vertical planar 

15 surface in the three-dimensional computer model is 
defined with a base defined by transformed corner points 
from the given camera which lie within the predetermined 
distance of the corresponding transformed corner points 
from the other camera, and with an aspect ratio 

20 corresponding to the aspect ratio of the planar surface 
in the image data of the given camera to which the 
transformed corner points belong. 

160. Apparatus according to claim 150, further comprising 
25 means for generating image data by rendering an image of 

the modelled object, in which texture data based on the 
identified image data from at least one camera is 
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rendered onto the model . 

161. Apparatus according to claim 160, operable to 
perform processing such that image data enclosed by each 

5 planar surface is rendered on the corresponding planar 
surface of the object model. 

162. Apparatus according to claim 160, further comprising 
means for displaying an image of the object using the 

10 generated image data. 

163. Apparatus for generating a model of an object in a 
three-dimensional computer model by processing images of 
the object from a plurality of cameras, the apparatus 

15 being operable to process image data from a first camera 
and a second camera to match feature points in the image 
data from the first camera with feature points in the 
image data from the second camera, to use the resulting 
matches to determine planar surfaces making up the 

20 object, and to represent the object in the computer model 
in dependence thereon . 

' ltW. A storage medium storing instructions for causing 

a prdsnrammable processing apparatus to perform a method 
25 according to - any of clainft^ 133 - to 14Q - , 

165. A signal \conveying instructions for causing a 
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>rogrammable processing apparatus to perform a method 
pi according to any o£ claiitf^ 133 to 14 0 . 

166. A method of processing image data defining a 
5 sequence^ of images of at least one object moving in a 
scene to produce signals defining a representation of 
each object\in a three-dimensional computer model, and 
to generate Mage data by rendering an image of the 
three-dimensionad computer model in accordance with a 
10 user-selected viewing direction, the method comprising: 

processing the\image data to identify image data 
relating to respective^ objects in the scene; 

defining a representation of each object in the 
three-dimensional computer\model in dependence upon the 
15 identified image data; 

generating image data by Vendering an image of the 
three-dimensional computer model in accordance with a 
user-selected viewing direction rti which texture data 
based on the identified image data i\ rendered onto the 
20 object representations; and 

generating quality information for ^the image data 
indicating a quality of the image data determined in 
dependence upon the user-selected viewing direction. 



25 



167. A m^hod according to claim 166, wherein the step 
of generatinSsouality information for the image includes 
generating information indicating the reliability of the 



• 
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image da&a in dependence upon the angle between the user- 
selected viewing direction and the viewing direction from 
which the input image data was recorded. 

5 F68 . A method according to claim 167, wherein the 
information indicating the reliability is generated in 
dependence upon a linear relationship between quality and 
the angujsar difference between the user-selected viewing 
direction aWl the viewing direction from which the input 
10 image data wak recorded . 



169. A method according to claim 167, further comprising 
the step of generating information indicating how to 
change the viewing direction to improve the generated 

15 reliability. \ 

170. A method according to claim 166, wherein image data 
for a sequence of images recorded by a first camera and 
a sequence of images recorded by a second camera are 

20 processed such that: 

in the step of processing the image data, image data 
from the first camera relating to the respective objects 
in the scene is identified, and image data from the 
second camera relating to the respective objects in the 
25 scene is identified; 

in the step of defining a representation of each 
object, a first representation of each object is defined 
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in dependence upon the identified image data from the 
first camera, and a second representation of each object 
is defined in dependence upon the identified image data 
from the second camera; and 
5 in the step of generating image data, texture data 

based on the identified image data from at least one 
camera is rendered onto the object representations. 

171. A method according to claim 166, wherein, in the 
10 step of defining a representation of each object, each 
object is represented as a planar surface. 

1a2. A method according to claim 166, wherein the quality 
information is generated as pixel data within the 
15 generated image data. 

173. A methoa\according to claims 166, further comprising 
the step of generating a signal conveying the image data 
and the quality information. 

20 

174. A method according to claim 173, further comprising 
the step of recording the signal. 



25 



1>5. A method according to claim 166, further comprising 
the su^p of displaying an image using the generated image 
data and displaying the quality information. 
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1^6. A method according to claim 166, further comprising 
tha step of making a recording of the image data and the 
qualVty information either directly or indirectly. 



\ 5 177, A method of rendering an image in accordance with 

a user-selected viewing direction of a three-dimensional 
computer model comprising a representation and associated 
texture dataV for at least one object, the texture data 
being derived\from image data recorded by at least one 



10 camera, the method comprising: 

generating i^mage data by rendering an image of the 
il \^o: 

user-selected viewing direction, in which the texture 



three-dimensional ^computer model in accordance with a 



data is rendered ontox^each representation; and 
15 generating qualityv information for the image data 

indicating a quality of\ the image data determined in 
dependence upon the user-selected viewing direction. 



178. An image processing method in which object data 
20 defining a three-dimensional computer model of at least 
one object in a scene is rendered in accordance with a 
user-selected viewing direction using image data recorded 
by a camera to render each object, \nd an indicator of 
a quality of the generated image dat?^ is produced for 
25 output to the user. 



179. Apparatus for processing image data^ defining a 
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sequence of images of at least one object moving in a 
scenes, to produce signals defining a representation of 
each ocrject in a three-dimensional computer model, and 
to generate image data by rendering an image of the 
three-dimensional computer model in accordance with a 
user-selectesfl viewing direction, the apparatus 
comprising : 

means for\ processing the image data to identify 
image data relating to respective objects in the scene; 

means for defining a representation of each object 
in the three-dimensional computer model in dependence 
upon the identified image data; 

means for generating image data by rendering an 

image of the three-dimensional computer model in 

accordance with a user-seYected viewing direction in 

which texture data based on the identified image data is 

\ 

rendered onto the object representations; and 

means for generating quali\ty information for the 
image data indicating a qualitv of the image data 
determined in dependence upon the user-selected viewing 
direction . 



180. Apparatus according to claim 179, wherein the means 
for generating quality information for the image 
comprises means (ror generating information indicating the 
reliability of tlie\mage data in dependence upon the 
angle between the user-sei^ected viewing direction and the 
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viewing auction from which the input image data was 
recorded , 



10 



H. Apparatus according to claim 180, operable to 
perform processing such that the information indicating 
the reliability is generated in dependence upon a linear 
relationship between quality and the angular difference 
between \he user-selected viewing direction and the 
viewing direction from which the input image data was 
recorded . 



182. Apparatus according to claim 180, further comprising 
means for generating\nf ormation indicating how to change 
the viewing direction to improve the generated 

15 reliability. ^^^^ 

183. Apparatus according to claim 179, operable to 
process image data for a sequence of images recorded by 
a first camera and a sequence of images recorded by a 

20 second camera, wherein: 

the means for processing the image data is operable 
to identify image data from the first camera relating to 
the respective objects in the scene and to identify image 
data from the second camera relating to the respective 
25 objects in the scene; 

the means for defining a representation of each 
object is operable to define a first representation of 
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each object in dependence upon the identified image data 
from the first camera, and to define a second 
representation of each object in dependence upon the 
identified image data from the second camera; and 
5 the means for generating image data is operable to 

render texture data based on the identified image data 
from at least one camera onto the object representations. 

184. Apparatus according to claim 179, wherein the means 
10 for defining a representation of each object is arranged 

to represent each object as a planar surface. 

185. Apparatus according to claim 179, operable to 
perform processing such that the quality information is 

15 generated as pixel data within the generated image data. 



186. Apparatus^ccording to claim 179, further comprising 
means for displaying an image using the generated image 
data and displaying the quality information. 



20 



187. Apparatus for rendering an image in accordance with 
a user-selected viewings^direction of a three-dimensional 
computer model comprisingva representation and associated 
texture data for at least\pne object, the texture data 
25 being derived from image data recorded by at least one 
camera, the apparatus comprising: 

means for generating ima^e data by rendering an 



130 

i\nage of the three-dimensional computer model in 
accordance with a user-selected viewing direction, in 
which the texture data is rendered onto each 
representation ; and 
5 me^ns for generating quality information for the 

image da^a indicating a quality of the image data 
determined\ in dependence upon the user-selected viewing 
direction , 

10 188- An image processing apparatus operable to render 

\ 

object data defining a three-dimensional computer model 
of at least one object in a scene in accordance with a 
user-selected viewing direction using image data recorded 



25 



by a camera to render each object, and operable to 

\ 

15 produce an indicator of a quality of the generated image 



data for output to the user. 

\ 

189. A storage medium storing instructions for causing 

\ 

a programmable processing apparatus to perform a method 

\ 

20 according to ^any of claim^ 166 to 170 . 

\ 

190. A signal conveying instructions for causing a 
programmable processing apparatus\ to perform a method 

\ 

p> according to -any of claim^ 166 -to 1 7, 0 



191. A method of processing image data defining a 
plurality of sequences of images, each from a respective 
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caViera, of at least one object moving in a scene to 

produce signals defining a representation of each object 

in a\ three-dimensional computer model, and to generate 

image \iata by rendering an image of the three-dimensional 

5 computeV model in accordance with a user-selected viewing 

direction, the method comprising: 

processing input image data from at least one camera 

to define at least one representation of each object in 

the three-dimensional computer model; and 

10 generating image data by rendering an image of the 

three-dimensional computer model in accordance with the 

\ 

user-selected viewing direction, in which texture data 
based on inputX image data is rendered onto a 
representation of each object; 
15 wherein: 

the representation of each object rendered is 
determined in dependence upon the user-selected viewing 

A 

direction, the respective viewing directions of cameras, 
and at least one camera \characteristic affecting image 
20 data quality. 

192. A method according to claim 191, wherein the 
representation of each ob jectXrendered is determined in 

dependence upon the user-selected viewing direction, the 

\ 

25 viewing direction of respective cameras, and at least one 
of: the methods of transf erring\ the image data from 
respective cameras; the resolution ot respective cameras; 



\ 
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the shutter speed of respective cameras; the stability 
of the image, data from respective cameras; and whether 
the image data\rom respective cameras is colour or black 
and white. 



10 



. A method according to claim 191, wherein the user- 
selected viewing direction is input prior to the step of 
definings^the object representations, and wherein one 
representation of each object is defined using the image 
data from one camera, the one camera being selected in 
dependence upon th\user-selected viewing direction, the 
viewing direction of rfegp^ctive cameras, and at least one 
camera characteristic 4f fating image data quality. 



15 194. A method according to claiVl91, wherein image data 
from a first camera is processed to define a first 
representation of each object in th\ three-dimensional 
computer model, image data from a second camera is 
processed to define a second representation of each 

20 object in the three-dimensional computer ^^pdel, and 
either the first representations or the\ second 
representations are selected for rendering in dependence 
upon the user-selected viewing direction, the vie\ing 
direction of the first and second cameras, and at leas 

25 one camera characteristic affecting image data quality, 



195. A method according to claim 191, wherein a plurality 
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of camera characteristics affecting image quality are 
considered to determine the representation of each object 
for rendering. 

5 196. A method according to claim 195, wherein the camera 
characteristics affecting quality are considered in a 
predetermined order and values for each respective camera 
characteristic are compared , with the determination of 
the representations to be rendered being made once the 
10 tests identify a characteristic which differs by more 
than a predetermined amount for given cameras . 

197. A method according to claim 191, further comprising 
the step of generating a signal conveying the image data. 

15 

198. A method according to claim 197, further comprising 
the step of recording the signal. 

199. A method according to claim 191, further comprising 
20 the step of displaying an image of the objects using the 

generated image data . 

200. A method according to claim 191, further comprising 
the step of making a recording of the image data either 

25 directly or indirectly. 



201. An image processing method in which image data from 
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eaAh of a respective sequence of images , each from a 
diffVrent camera, is processed to define a representation 
of atyeast one object in a three-dimensional computer 
model, a^nd wherein a representation of each object is 
selected f^pr rendering in dependence upon a user-selected 
viewing direction, the viewing direction of each camera 
and at least \pne camera parameter related to image data 
quality . 




10 202. An image processing method in which a user-selected 
viewing direction in accordance with which an image of 
at least one object in\a three-dimensional computer model 



is to be rendered is used to select, from among image 
data defining a plurality of images of the object each 



15 recorded by a respective camera, image data to be used 
to define the object in the\three-dimensional computer 
model , the selection being carried out in dependence upon 
the user-selected viewing direction, together with the 

\ 

viewing direction of each camera and at least one camera 
20 parameter related to image data quality. 



203. An image processing apparatus f oAprocessing image 
data defining a plurality of sequences of images, each 
from a respective camera, of at least one\^bject moving 
25 in a scene to produce signals defining a representation 
of each object in a three-dimensional computer^model , and 
to generate image data by rendering an image, of the 
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th:£\ee-dimensional computer model in accordance with a 
usereselected viewing direction, the apparatus 
comprising : 

means for processing input image data from at least 
5 one camera to define at least one representation of each 
object im the three-dimensional computer model; and 

means\ for generating image data by rendering an 
image of the three-dimensional computer model in 
accordance wmh the user-selected viewing direction, in 
10 which texture oata based on input image data is rendered 
onto a representation of each object; 



the apparatus being operable to perform processing 
such that: \ 

the representation of each object rendered is 
15 determined in dependence upon the user-selected viewing 
direction, the respective viewing directions of cameras, 



and at least one camera characteristic affecting image 
data quality, 

20 204. Apparatus according Vo claim 203, operable to 

perform processing such tha^the representation of each 

object rendered is determined in dependence upon the 

user-selected viewing direction, the viewing direction 

of respective cameras, and at least one of: the methods 

\ 

25 of transferring the image data from respective cameras; 
the resolution of respective cameras; the shutter speed 
of respective cameras; the stability, of the image data 
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)m respective cameras; and whether the image data from 
respective cameras is colour or black and white. 



10 



15 



20 



25 



[5- Apparatus according to claim 203, operable to 
perform processing such that, when the user-selected 
viewing\ direction is input prior to the object 
representations being defined, one representation of each 
object is def in&dTNising the image data from one camera, 
the one camera Vbeing selected in dependence upon the 
user-selected viewing^^irection, the viewing direction 
of respective cameras, ^and at least one camera 
characteristic affecting imageNlata quality. 



peri 



• Apparatus according to claim 203, operable to 
processing such that image data from a first 
camera fs processed to define a first representation of 
each objectT\in the three-dimensional computer model, 
image data f roin^a second camera is processed to define 
a second represent4^€*n of each object in the three- 
dimensional computer mbdel, and either the first 
representations or the second representations are 
selected for rendering in dependence upon the user- 
selected viewing direction, the viewing direction of the 
first and second cameras, and at le^st one camera 
characteristic affecting image data quality 



207 




aratus according to claim 203, operable to 
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jrform processing such that a plurality of camera 
characteristics affecting image quality are considered 
to determine the representation of each object for 
rendering 



10 



208. Apparatus \according to claim 207, operable to 
perform processing\such that the camera characteristics 
affecting quality are\pnsidered in a predetermined order 
and values for each respe^rtive camera characteristic are 
compared, with the detearmin^tion of the representations 
to be rendered being made orK>e the tests identify a 
characteristic which differs by more^than a predetermined 
amount for given cameras . 



15 209. Apparatus according to claim 203, further comprising 
means for displaying an image of the objects using the 
generated image data • 



25 



z\U) . An image processing apparatus operable to process 
image\data from each of a respective sequence of images, 
each fronNa different camera, to define a representation 
of at least ohe object in a three-dimensional computer 
model, and to select a representation of each object for 
rendering in dependence upon a user-selected viewing 
direction, the viewing dir^^tion of each camera and at 
least one camera parameter relatecK£o image data quality. 
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211. \An image processing apparatus operable to use a 
user-selected viewing direction in accordance with which 
an image of at least one object in a three-dimensional 
computer\ model is to be rendered to select, from among 

5 image datek defining a plurality of images of the object 
each recorded by a respective camera, image data to be 
used to defjLne the object in the three-dimensional 
computer modeli, the selection being carried out in 
dependence upon\ the user-selected viewing direction, 
10 together with the\ viewing direction of each camera and 
at least one camera parameter related to image data 
quality. 

212. A storage medium sVoring instructions for causing 
15 a programmable processing\apparatus to perform a method 

pN according to -«*y=©# claim\\l91 to 303 . 

213. A signal conveying instructions for causing a 
programmable processing apparatus to perform a method 

20 according to - any - of claim^ 191 ^o<^202 « . 

214. A method of processing image data defining a 
plurality of sequences of images, each from a respective 
camera, of an object moving in a scene to produce signals 

25 defining a representation of the object in a three- 
dimensional computer model, and to generate image data 
for first and second images in a sequence of images of 
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the object by rendering images of the three-dimensional 
computer model in accordance with first and second user- 
selected viewing directions, the method comprising: 

processing the image data to define at least one 
5 representation of the object in the three-dimensional 
computer model; 

generating image data for use in a first image in 
the sequence by rendering texture data based on image 
data from at least a first of the cameras onto a 
10 representation of the object in accordance with a first 
user-selected viewing direction; 

generating image data for use in a second image in 
the sequence by rendering texture data based on image 
data from a second of the cameras onto a representation 
15 of the object in accordance with a second user-selected 
viewing direction; 

testing whether first and second images of the 
object displayed from the generated image data will be 
discontinuous by testing whether the image data for the 
20 object in the second image in the sequence differs by 
more than a predetermined amount from predetermined image 
data ; and 

if the image data for the object in the second image 
differs by more than the predetermined amount, generating 
25 modified image data for the object in the second image. 

215. A method according to claim 214, wherein: 
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in the step of processing the image data, image data 
from the first camera is processed to generate a first 
representation of the object in the three-dimensional 
computer model, and image data from the second camera is 
processed to generate a second representation of the 
object in the three-dimensional computer model; 

the image data for the object in the first image is 
generated by rendering the first representation; and 

the image data for the object in the second image 
is generated by rendering the second representation. 

216. A method according to claim 215, wherein, in the 
step of testing: 

further image data for the object in the second 
image in the sequence is generated by rendering texture 
data based on image data from the first camera onto the 
first representation of the object in accordance with the 
second user-selected viewing direction; and 

the image data for the object in the second image 
generated using image data from the second camera is 
compared with the image data for the object in the second 
image generated using image data from the first camera, 

217, A method according to claim 216, wherein in the step 
of generating modified image data, the modified image 
data is generated in dependence upon the image data for 
the object in the second image generated using image data 



from the second camera and the image data for the object 
in the second image generated using image data from the 
first camera . 

218. A method according to claim 214, wherein: 

the step of generating image data for the first 
image comprises rendering the three-dimensional computer 
model in accordance with the first user-selected viewing 
direction; 

the step of generating image data for the second 
image comprises rendering the three-dimensional computer 
model in accordance with the second user-selected viewing 
direction; and 

the step of testing comprises comparing the rendered 
image data for the second image with the predetermined 
image data . 

219. A method according to claim 214, further comprising 
the step of generating a signal conveying the modified 
image data . 

220. A method according to claim 219, further comprising 
the step of recording the signal. 

221. A method according to claim 214, further comprising 
the step of displaying an image of the object using the 
modified image data. 
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222. A method according to claim 214, further comprising 
the step of making a recording of the modified image data 
either directly or indirectly. 

5 223. A method of generating image data for first and 
second images in a sequence of images by rendering a 
three-dimensional computer model in accordance with 
respective first and second user-selected viewing 

£3 directions, the three-dimensional computer model 

fy 10 comprising a representation and associated texture data 
for at least one object and the texture data comprising 

;S texture data derived from image data recorded by a first 

camera and texture data derived from image data recorded 

.-"f by a second camera, the method comprising: 

15 generating image data for use in a first image in 

^3 the sequence by rendering texture data based on image 

data from at least a first camera onto the representation 
of each object in accordance with the first user-selected 
viewing direction ; 
20 generating image data for use in the second image 

in the sequence by rendering texture data based on image 
data from the second camera onto the representation of 
each object in accordance with a second user-selected 
viewing direction ; 
25 testing whether first and second images of the 

object displayed from the generated image data will be 
discontinuous by testing whether the image data for the 
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object in the second image in the sequence differs by 
more than a predetermined amount from predetermined image 
data; and 

if the image data for the object in the second image 
5 differs by more than the predetermined amount, generating 
modified image data for the object in the second image, 

224. An image processing method in which a three- 
r3 dimensional computer model including at least one 
_Fy 10 representation of an object is processed a first time to 
[q generate image data for a first image in a sequence of 
r£ images by rendering using image data recorded by a first 

camera as the basis for texture data for a 
^ representation, and a second time to generate image data 

15 for a successive image in the sequence by rendering using 
^9 image data recorded by a second camera as the basis for 

. r~z 

texture data for a representation, and modified image 
data is generated for the object in the successive image 
if image data comparison tests indicate that the object 
20 in the images in the sequence will appear discontinuous. 

225. A method of generating image data for successive 
images in a sequence by rendering a representation of an 
object in a three-dimensional computer model using image 

25 data from a plurality of cameras, in which a test on the 
image data is performed to determine whether the image 
of the object will appear discontinuous in the successive 
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images, and the image data is processed to reduce the 
discontinuity. 

226 . Image processing apparatus for processing image data 
5 defining a plurality of sequences of images, each from 
a respective camera, of an object moving in a scene to 
produce signals defining a representation of the object 
in a three-dimensional computer model, and to generate 
=« image data for first and second images in a sequence of 

if! 10 images of the object by rendering images of the three- 
's dimensional computer model in accordance with first and 
second user-selected viewing directions, the apparatus 
^ comprising: 

J*f means for processing the image data to define at 

H; 15 least one representation of the object in the three- 

. — 

_.: — 

\Q dimensional computer model; 

means for generating image data for use in a first 
image in the sequence by rendering texture data based on 
image data from at least a first of the cameras onto a 
20 representation of the object in accordance with a first 
user-selected viewing direction; 

means for generating image data for use in a second 
image in the sequence by rendering texture data based on 
image data from a second of the cameras onto a 
25 representation of the object in accordance with a second 
user-selected viewing direction; 

means for testing whether first and second images 
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of the object displayed from the generated image data 
will be discontinuous by testing whether the image data 
for the object in the second image in the sequence 
differs by more than a predetermined amount from 
5 predetermined image data; and 

means for generating modified image data for the 
object in the second image if the image data for the 
object in the second image differs by more than the 
predetermined amount, 

10 

227. Apparatus according to claim 226, wherein: 

the means for processing the image data is operable 
to process image data from the first camera to generate 
a first representation of the object in the three- 
15 dimensional computer model, and to process image data 
from the second camera to generate a second 
representation of the object in the three-dimensional 
computer model; and 

the apparatus is arranged to perform processing such 

20 that: 

- the image data for the object in the first image 
is generated by rendering the first representation; and 

- the image data for the object in the second image 
is generated by rendering the second representation. 



228. Apparatus according to claim 227, wherein the means 
for testing is operable to perform processing such that: 
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further image data for the object in the second 
image in the sequence is generated by rendering texture 
data based on image data from the first camera onto the 
first representation of the object in accordance with the 
5 second user-selected viewing direction; and 

the image data for the object in the second image 
generated using image data from the second camera is 
compared with the image data for the object in the second 
f « image generated using image data from the first camera. 

S^= 5 

J5 229. Apparatus according to claim 228, wherein the means 

for generating modified image data is arranged to perform 

^ processing such that the modified image data is generated 

^ in dependence upon the image data for the object in the 

15 second image generated using image data from the second 

%S camera and the image data for the object in the second 

image generated using image data from the first camera. 

230. Apparatus according to claim 226 , wherein: 
20 the means for generating image data for the first 

image comprises means for rendering the three-dimensional 
computer model in accordance with the first user-selected 
viewing direction ; 

the means for generating image data for the second 
25 image comprises means for rendering the three-dimensional 
computer model in accordance with the second user- 
selected viewing direction; and 
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the means for testing comprises means for comparing 
the rendered image data for the second image with the 
predetermined image data • 

231, Apparatus according to claim 226, further comprising 
means for displaying an image of the object using the 
modified image data. 

232. Image processing apparatus for generating image data 
for first and second images in a sequence of images by 
rendering a three-dimensional computer model in 
accordance with respective first and second user-selected 
viewing directions, the three-dimensional computer model 
comprising a representation and associated texture data 
for at least one object and the texture data comprising 
texture data derived from image data recorded by a first 
camera and texture data derived from image data recorded 
by a second camera, the apparatus comprising: 

means for generating image data for use in a first 
image in the sequence by rendering texture data based on 
image data from at least a first camera onto the 
representation of each object in accordance with the 
first user-selected viewing direction; 

means for generating image data for use in the 
second image in the sequence by rendering texture data 
based on image data from the second camera onto the 
representation of each object in accordance with a second 
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user-selected viewing direction; 

means for testing whether first and second images 
of the object displayed from the generated image data 
will be discontinuous by testing whether the image data 
for the object in the second image in the sequence 
differs by more than a predetermined amount from 
predetermined image data; and 

means for generating modified image data for the 
object in the second image if the image data for the 
object in the second image differs by more than the 
predetermined amount . 

233. An image processing apparatus operable to process 
a three-dimensional computer model including at least one 
representation of an object a first time to generate 
image data for a first image in a sequence of images by 
rendering using image data recorded by a first camera as 
the basis for texture data for a representation, and a 
second time to generate image data for a successive image 
in the sequence by rendering using image data recorded 
by a second camera as the basis for texture data for a 
representation, and operable to generate modified image 
data for the object in the successive image if image data 
comparison tests indicate that the object in the images 
in the sequence will appear discontinuous. 



234. Apparatus for method of generating image data for 
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successive images in a sequence by rendering a 
representation of an object in a three-dimensional 
computer model using image data from a plurality of 
cameras, the apparatus being operable to perform a test 
on the image data to determine whether the image of the 
object will appear discontinuous in the successive 
images, and to process the image data to reduce the 
discontinuity. 

10 335 • A storage medium storing instructions for causing 
a programmable processing apparatus to perform a method 
^ accordsLng to - a - ny of claim\ 214^ feo 225 . 

236. A signal conveying instructions for causing a 
15 programmable processing apparatus to perform a method 

according to any ^| claim\ 214 to 22 5 » . 

237. An linage processing apparatus for processing image 
data defining a plurality of seguences of images, each 

20 from a respective camera, of a plurality of objects 
moving in a scene to produce signals defining 
representations of\the objects in a three-dimensional 
computer model, the apparatus comprising: 

an image data identifier for processing image data 

25 from a first of the cameras to identify image data 
relating to objects in the ^scene, and for processing 
image data from a second of the \ameras to identify image 




-.ft 



10 



15 
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data delating to objects in the scene; 

aA object representation def iner for processing the 
identified image data from the first camera for each 
object tovdefine an object representation in a modelling 
space having a height dependent upon the image data for 
the object ftrom the first camera, and for processing the 
identified image data from the second camera for each 
object to define an object representation in the 
modelling space having a height dependent upon the image 
data for the ob\ject from the second camera; 

a height comparer for comparing the height of the 
representation of each object generated in dependence 
upon image data fitom the first camera with the height of 
the representation \of the corresponding object generated 
in dependence upon jWage data from the second camera; and 
an object representation generator for generating 
object representations in the three-dimensional computer 
model in dependence upon the height comparisons. 



20 238. Apparatus for processing image data defining a 
plurality of sequences of images, each from a respective 
camera, of an object moving in a scene to produce signals 
defining a representation of the object in a three- 
dimensional computer model, the apparatus comprising: 

25 an image data identifier for processing image data 

from a first of the cameras to identify image data 
relating to the object in the scene, and for processing 
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image data from a second of the cameras to identify image 

data relating to the object in the scene; 

an image data transformer for applying a 

transformation to the identified image data from the 
5 first camera which defines a mapping from the ground 

plane in the space of the image data of the first camera 

to a surface' in a modelling space, and for applying a 

transformation to the identified image data from the 

second camera which defines a mapping from the ground 
10 plane in the space of the image data of the second camera 

to the surface in the modelling space; 

an image data comparer for comparing the trans formed 

image data from the first and second cameras on the 

surface in the modelling space; 
15 a shadow determinator for determining which part of 

the image data represents shadow in dependence upon the 

comparison results; and 

an object representation generator for generating 

a representation of at least the object in the three- 
20 dimensional model. 

^239 ♦ Apparatus for generating a model of an object in a 
threfe^dimensional computer model , comprising: 

an >i,mage data transformer for applying a 
25 transformations^) image data from a first camera relating 
to the object and\ts shadow which maps the image data 
for one of the object a*id its shadow to a surface, and 



m 
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ror applying a transformation to image data from a second 
camera relating to the object and its shadow which maps 
the ima^e data for one of the object and its shadow to 
the surf ace^cmd 
5 an ob jectNuodeller for modelling the object in 

dependence upon part^^f the transformed image data. 

240. Apparatus for processing image data defining a 
plurality of sequences of images , each from a respective 

10 camera, of an object moving in a scene to produce signals 
defining a representation of the object in a three- 
dimensional computer model, the apparatus comprising: 

an image data identifier for processing image data 
from a first of the cameras to identify image data 

15 relating to the object in the scene, and for processing 
image data from a second of the cameras to identify image 
data relating to the object in the scene; 

a footprint determinator for processing the 
identified image data from the first camera and the 

20 identified image data from the second camera to determine 
a footprint of the object on the ground; and 

an object modeller for defining a model of the 
object in the three-dimensional computer model in 
dependence upon the determined footprint. 

25 

241. Apparatus for processing image data defining a 
sequence of images of a plurality of objects moving in 
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a scene to produce signals defining representations of 
the objects in a three-dimensional computer model, and 
to generate image data by rendering an image of the 
three-dimensional computer model in accordance with a 
5 user-selected viewing direction, the apparatus 
comprising: 

an image data identifier for processing the image 
data to identify image data relating to respective 
C3 objects in the scene; 

fit 10 an object modeller for defining a representation of 

ty each object in the three-dimensional computer model, in 

lQ dependence upon the identified image data; and 

~~" a renderer for generating image data by rendering 

12 an image of the three-dimensional computer model in 

V 15 accordance with a user-selected viewing direction, 
operable such that, when the selected viewinq direction 
is within a predetermined range of viewing directions, 
texture data based on the identified image data is 
rendered onto the object representations, and, when the 
20 selected viewing direction is not within the 
predetermined range of viewing directions, a schematic 
of the positions of the objects in the scene is rendered. 



24 2. Apparatus for rendering an image in accordance with 
25 a user-selected viewing direction of a three-dimensional 
computer model comprising a representation and associated 
texture data for an object, the texture data being 
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quality information generator for generating 



.V 



quality\ information for the image data indicating a 
quality df the % image data determined in dependence upon 
the user-selected viewing direction. 

245. ApparatusXf or rendering an image in accordance with 
a user-selected Viewing direction of a three-dimensional 
computer model comprising a representation and associated 
texture data for at\ least one object, the texture data 
being derived from imiage data recorded by at least one 
camera, the apparatus comprising: 

a renderer for generating image data by rendering 
an image of the three-dimensional computer model in 
accordance with a user-selea^ed viewing direction, in 
which the texture data is rendered onto each 
representation ; and 

a quality data generator foV generating quality 
information for the image data indicating a quality of 
the image data determined in dependence upon the user- 
selected viewing direction. 



246. An image processing apparatus for processing image 
data defining a plurality of sequences of imkges, each 
from a respective camera, of at least one objecrt moving 
in a scene to produce signals defining a representation 
of each object in a three-dimensional computer model\ and 
to generate image data by rendering an image of the 
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derived from image data recorded by at least one camera, 
the apparatus comprising: 

a first renderer for rendering the texture data onto 
the representation for the object in accordance with the 
5 user-selected viewing direction when the user-selected 
viewing direction is within a predetermined range of 
viewing directions; and 

a second renderer for rendering a schematic of the 
positions of the object when the user-selected viewing 
10 direction is not within the predetermined range of 
viewing directions . 

243. Apparatus for processing image data defining a 
plurality of sequences of images, each from a respective 

15 camera, of an object moving in a scene to produce signals 
defining a representation of the object in a three- 
dimensional computer model, the apparatus comprising: 

an image data identifier for processing image data 
from a first of the cameras to identify image data 

20 relating to the object in the scene, and for processing 
image data from a second of the cameras to identify image 
data relating to the object in the scene; 

a surfacer identifier for processing the identified 
image data from the first camera and the identified image 

25 data from the second camera to identify planar surfaces 
on which points on the object lie, comprising a feature 
matcher for matching feature points in the identified 
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image data from the first camera with feature points in 
the identified image data from the second camera, and a 
planar surface identifier for identifying planar surfaces 
on which matched feature points lie; and 

an object modeller for defining a model of the 
object in the three-dimensional computer model in 
dependence upon the identified planar surfaces . 

t4. Apparatus for processing image data defining a 
10 sequence of images of at least one object moving in a 
scene to produce signals defining a representation of 
each object in a three-dimensional computer model, and 
to generate image data by rendering an image of the 
three-dimensional computer model in accordance with a 
15 user-selected^ viewing direction, the apparatus 
comprising: 

an image dat^, identifier for processing the image 
data to identify image data relating to respective 
objects in the scene; 

20 an object modeller for defining a representation of 

each object in the three-dimensional computer model in 
dependence upon the identif ied^image data; 

a renderer for generating image data by rendering 
an image of the three-dimensional computer model in 

25 accordance with a user-selected viewing direction in 
which texture data based on the identified image data is 
rendered onto the object representationsV and 



20 
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th3?ee-dimensional computer model in accordance with a 
userVselected viewing direction, the apparatus 
comprising : 

an\object representation generator for processing 
5 input image data from at least one camera to define at 
least one \representation of each object in the three- 
dimensional V:omputer model; and 

a renderer for generating image data by rendering 
an image of uhe three-dimensional computer model in 
10 accordance with the user-selected viewing direction, in 
which texture data based on input image data is rendered 
onto a representation of each object; 

the apparatus beting operable to perform processing 
such that: 

15 the representation^ of each object rendered is 

determined in dependence upon the user-selected viewing 
direction, the respective Viewing directions of cameras, 
and at least one camera characteristic affecting image 



data quality. 



247. Image processing apparatus for processing image data 
defining a plurality of sequences of images, each from 
a respective camera, of an object moving in a scene to 
produce signals defining a representation of the object 
25 in a three-dimensional computer model, and to generate 
image data for first and second images in a sequence of 
images of the object by rendering images of the three- 
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dimensional computer model in accordance with first and 
second user-selected viewing directions, the apparatus 
comprising : 

an object representation generator for processing 
5 the image data to define at least one representation of 
the object in the three-dimensional computer model; 

a renderer for generating image data for use in a 
first image in the sequence by rendering texture data 
based on image data from at least a first of the cameras 
10 onto a representation of the object in accordance with 
a first user-selected viewing direction, and for 
generating image data for use in a second image in the 
sequence by rendering texture data based on image data 
from a second of the cameras onto a representation of the 
15 object in accordance with a second user-selected viewing 
direction; 

an image data tester for testing whether first and 
second images of the object displayed from the generated 
image data will be discontinuous by testing whether the 

20 image data for the object in the second image in the 
sequence differs by more than a predetermined amount from 
predetermined image data; and 

an image data modifier for generating modified image 
data for the object in the second image if the image data 

25 for the object in the second image differs by more than 
the predetermined amount . 
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248* Image processing apparatus for generating image data 
for first and second images in a sequence of images by 
rendering a three-dimensional computer model in 
accordance with respective first and second user-selected 
5 viewing directions, the three-dimensional computer model 
comprising a representation and associated texture data 
for at least one object and the texture data comprising 
texture data derived from image data recorded by a first 
£3 camera and texture data derived from image data recorded 

fy *P b y a secon d camera, the apparatus comprising: 

IjS a renderer for generating image data for use in a 

CD 

yg first image in the sequence by rendering texture data 

based on image data from at least a first camera onto the 
f7 representation of each object in accordance with the 

r JT 15 first user-selected viewing direction, and for generating 
^ image data for use in the second image in the sequence 

by rendering texture data based on image data from the 
second camera onto the representation of each object in 
accordance with a second user-selected viewing direction; 
20 an image data tester for testing whether first and 

second images of the object displayed from the generated 
image data will be discontinuous by testing whether the 
image data for the object in the second image in the 
sequence differs by more than a predetermined amount from 
25 predetermined image data; and 

an image data modifier for generating modified image 
data for the object in the second image if the image data 
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for the object in the second image differs by more than 
the predetermined amount . 



